{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if you need to install the statsmodel library\n",
    "!pip install --user statsmodel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if you need to install scikit-learn\n",
    "!pip install --user sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 6 - Review of linear regression\n",
    "\n",
    "A linear relationship between two variables is one in which the scatterplot of them looks roughly like a line.  *Linear regression* is a method for modelling how a *dependent variable* linearly depends on one or more *independent variables*.  The dependent variable (also called a *response variable* and many other things) is what we are trying to model or predict, and is usually denoted $Y$.  The independent variables (also called *explanatory* or *input variables*) are the information we are using to make the predictiong, and are usually denoted $X_1, X_2, ...$.\n",
    "\n",
    "The linear relationship is: $$Y = \\beta_0 + \\beta_1 X_1 + \\beta_2 X_2 + ... + \\beta_n X_n + \\epsilon$$\n",
    "where $\\epsilon$ represents the error.\n",
    "\n",
    "Here, $Y, X_1, X_2, ..., X_n$ are *random variables* which is what mathematical variables that can take different values with different probabilities are called.\n",
    "\n",
    "Linear regression finds the coefficients $\\beta_0, \\beta_1, ..., \\beta_n$ so that the sum of the squares of the error term for each data observation is minimized (as small as possible).\n",
    "\n",
    "In this lab, we will look at a classic dataset used for studying regression containing data about Boston housing prices in the 1950s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import statsmodels.formula.api as smf\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the dataset from the sci-kit learn package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_boston\n",
    "boston_dict = load_boston()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the variable `boston_dict`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "boston_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`boston_orig` is not a dataframe!  It is something called a *dictionary* that contains the information in separate properties, called *keys*.  We can get the list of keys:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "boston_dict.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see what is linked to a key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "boston_dict.feature_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see the description of the data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(boston_dict.DESCR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happens if you leave off print?\n",
    "\n",
    "To convert the data into a Pandas dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "boston = pd.DataFrame(boston_orig.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the dataframe `boston`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are no column names, so let's repeat that command, but telling Pandas that the column names are in `feature_names`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "boston = pd.DataFrame(boston_orig.data, columns = boston_dict.feature_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the new dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each column in an independent variable.  (The columns were described when we did `print(boston_dict.DESCR)`).  The dependent variable that we want to predict is linked to the key called `target`, so we will add it to our dataframe as a column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "boston[\"price\"] = boston_dict.target "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the column was added correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at how a few of the variables relate to the median housing price.\n",
    "\n",
    "Let's see how the per capita crime rate per town (column `CRIM`) relates to median housing price (column `price` which is in $1000s) by plotting a scatter plot of the two variables.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details> <summary>Hint:</summary>\n",
    "    <code>df.plot.scatter(x = \"x_column_name\", y = \"y_column_name\")</code>\n",
    "</details>\n",
    "\n",
    "Make a scatter plot of the relationship between the average number of rooms per dwelling (column `RM`) and the price."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally plot the relationship between the pupil-teacher ratio per town (column `PTRATIO`) and the price."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of the three variable, `CRIM`, `RM`, and `PTRATIO`, which seems to have the most linear relationship with `price`?  Which of the relationships were positive (had a positive slope) and which were negative?\n",
    "\n",
    "Let's perform linear regression using `RM` (average number of rooms) as the independent variable.  We want to predict `price` our dependent variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lm = smf.ols('price ~ RM',boston).fit()\n",
    "lm.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's the equation of our linear model?\n",
    "\n",
    "We can visualize the line using Seaborn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.regplot(y=\"price\", x=\"RM\", data=boston, fit_reg = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *residuals* are the difference between the actual value of price and the value predicted by the regression line, for each row of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lm.resid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot a histogram of the residuals.  If the linear model is a good fit, the histogram should look like a normal distribution.  Does it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To better understand the relationship between the actual prices and the predicted or *fitted* values, we can plot a scatterplot of the two.  To get the fitted values, type `lm.fittedvalues`.  Try it below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot a scatter plot of with the price as the x variable and the fitted values as the y variable.  The code is slightly different since the fitted values aren't part of the `boston` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(boston['price'], lm.fittedvalues)\n",
    "plt.xlabel(\"Prices ($1000s)\")\n",
    "plt.ylabel(\"Predicted prices ($1000s)\")\n",
    "plt.title(\"Prices vs Predicted Prices\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If all the prices were predicted correctly, what would this plot have looked like?  By noticing where this plot is least like a plot with perfectly predicted prices, we can see where our linear model fails (makes bad predictions).  Where do you notice errors in the prediction?\n",
    "\n",
    "The most expensive houses are predicted incorrectly.  This is known as a *ceiling effect*, where the independent variable no longer has an effect on the dependent variable.\n",
    "\n",
    "Find the linear model showing how price depends on `PTRATIO`, the pupil-teacher ratio by town:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the fit of your model by looking at the histogram of the residuals and the scatter plot of the predicted prices vs. the actual prices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do you think this linear model is a good predictor of prices?\n",
    "\n",
    "Let's try making a linear regression model with three independent variables:\n",
    "\n",
    "- `CRIM` (per capita crime rate by town)\n",
    "- `RM` (average number of rooms per dwelling)\n",
    "- `PTRATIO` (pupil-teacher ratio by town)\n",
    "\n",
    "The code will be similar to above, but the formula is `price ~ CRIM + RM + PTRATIO`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the fit of your model by plotting the histogram of the residuals and the scatter plot of the predicted prices vs. the actual prices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How did this linear model perform?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}